Skip to content

[#901] Make scheduled report runs retryable without duplication - #909

Merged
greatest0fallt1me merged 1 commit into
Predictify-org:mainfrom
Baskarayelu:feat/issue-901-report-retries
Aug 27, 2026
Merged

[#901] Make scheduled report runs retryable without duplication#909
greatest0fallt1me merged 1 commit into
Predictify-org:mainfrom
Baskarayelu:feat/issue-901-report-retries

Conversation

@Baskarayelu

Copy link
Copy Markdown

Summary

Make scheduled report execution durable across queue redelivery, worker overlap, and process restarts. Each schedule period now has one database-backed run identity, an expiring owner lease, bounded retries, and an explicit terminal outcome.

What changed

  • Added scheduled_report_runs with a unique (scheduled_report_id, run_for) identity and indexes for ready/leased work.
  • Added atomic lease claims with owner tokens; stale leases can be recovered, while a previous worker cannot acknowledge output after ownership changes.
  • Added bounded exponential retry scheduling and visible retryable/failed states with truncated failure metadata.
  • Added a dedicated BullMQ queue/worker adapter and Prometheus counters for outcomes, retries, and lease conflicts.
  • Added regression tests for UTC identity normalization, overlapping workers, generator timeouts, restart/recovery through expired leases, retry exhaustion, lost ownership, invalid output, and worker lifecycle.
  • Fixed two pre-existing merge blockers encountered during validation: the duplicate prediction schema field and malformed user-prediction route block.

Acceptance criteria

  • A report period is generated once per schedule identity — database uniqueness plus queue identity.
  • Overlapping workers cannot duplicate output — conditional lease claim and owner-token guarded transitions.
  • Retry exhaustion is visible and bounded — attempt counter, retryable/failed states, capped backoff, and metrics.
  • Timeout, restart, overlap, and recovery behavior is covered by focused tests.

Validation

  • npm test -- --runInBand --forceExit tests/scheduledReportJobService.test.ts tests/scheduledReportWorker.test.ts
  • npx tsc --noEmit ... on the new service/worker modules

The repository-wide TypeScript build still reports unrelated pre-existing errors in other routes/services.

Security and failure modes

Completion and failure writes require the current lease token, stale workers cannot mutate a replacement worker's run, error text is bounded before persistence, and retry jobs use distinct IDs while retaining the same durable run identity.

Closes #901

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[GrantFox][Medium] Make scheduled reports retryable without duplication

2 participants